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ABSTRACT 

This report provides feedback on the implementation of 
America’s Choice school design in a Florida school district. It seeks 
evidence of the impacts of the design, focusing on the literacy workshops. 

The evaluation asks three central questions: Is America’s Choice being 
carried out in the manner envisioned? Are instructional practices of teachers 
changing in ways that would improve student learning? To what degree can 
improvements in student achievement be attributed to the design? To answer 
these questions, researchers compared the performance of students in 
America’s Choice to that of students in other schools in the Duval County 
school district in northeastern Florida. The data show that students in the 
America’s Choice schools regularly outgained students in other district 
schools in writing and, to a lesser extent, reading and mathematics. 
Detectable patterns of effect were strongest in writing; students in 
America’s Choice schools consistently demonstrated higher learning gains than 
students in other schools in the district. The results were most dramatic in 
fourth grade. Overall, across grades and subjects, the America’s Choice 
design had an equal effect on white and minority students. In 22 percent of 
the grades and subjects examined, America’s Choice was associated with a 
statistically significant reduction in the gap in performance between white 
and minority students by increasing the performance of minority students at a 
higher rate than white subjects. Appended are: (1) Writing Results by Year, 
Grade, and Cohort; (2) Reading Results by Year and Grade for Cohort II; (3) 
Reading and Mathematics Results by Year and Grade for Cohort III; (4) 
Predicted Writing Performance Gains for White and Minority Students in 
America’s Choice and Other District Schools; (5) Predicted Reading 
Performance Gains for White and Minority Students in America’s Choice and 
Other District Schools; and (6) Predicted Mathematics Performance Gains for 
White and Minority Students in America’s Choice and Other District Schools. 
(RJM) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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About the Americans 
Choice Design 

T he America’s Choice School Design 
is a K- 12 comprehensive school 
reform model designed by the 
National Center on Education and the 
Economy. America’s Choice focuses on raising 
academic achievement by providing a rigorous 
standards-based curriculum and safety net for all 
students. The goal of America’s Choice is to 
make sure that all but the most severely 
handicapped students reach an internationally 
benchmarked standard of achievement in 
English/language arts and mathematics by the 
time that they graduate. 

America’s Choice does not offer schools a script 
or a paint-by-numbers approach to reformed 
instruction. America’s Choice recognizes that 
the pace of change will vary from school to 
school and the model does not have a rigid 
implementation schedule. Rather, the core of the 
design contains a set of principles about the 
purpose of schooling and how schools should 
operate as well as a set of tools for building a 
program based on those principles. The essential 
principles and tools include: 

• High expectations for student performance 
that specify what students should know and 
be able to do at certain educational 
junctures. These standards are explicitly 
expressed through the New Standards 
Performance Standards that provide a 
common set of expectations for students and 
teachers. 

• An initial focus on literacy that features 
elements of phonics, oral language, shared 
books, guided and independent reading, 
daily writing, and independent writing. 

• A common core curriculum that is aligned 
with the standards. Through the America’s 
Choice literacy workshops. Core 
Assignments, and Foundations of Advanced 
Mathematics, school life is organized around 
a core curriculum. 



• Standards-based assessments, including 
the New Standards Reference Examination, 
that are aligned with the standards and the 
core curriculum, and that provide detailed 
feedback to teachers and students about 
student skill levels in relation to standards. 

t A distributed school leadership structure, 

led by the school’s principal, that 
coordinates implementation, analyzes results 
and sets performance targets, implements 
safety net programs to provide time for 
students to receive additional instruction, 
ensures the necessary resources, and aligns 
schedules and other school activities with 
implementation of the design. 

• Safety nets that are structured into the 
school day and year and that provide 
students with extensive support and multiple 
opportunities to achieve the standards. 

• A commitment to teacher professionalism 
that enables teachers to function as full 
professionals by providing ongoing, on-site 
professional development and support that is 
aligned with the standards and in which 
content and pedagogy are intimately 
connected. 

In order to become an America’s Choice school, 
over 80% of a school’s faculty must indicate 
their commitment to the America’s Choice 
design and agree to implement the program over 
three years. Each school must assign personnel 
as coaches to lead the implementation of the 
design, and a parent/community outreach 
coordinator who ensures that students get 
needed support services. 
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Evaluation of 
Americans Choice 

T he Consortium for Policy Research in 
Education (CPRE) at the University 
of Pennsylvania was contracted by 
the National Center on Education and 
the Economy (NCEE) to conduct the external 
evaluation of the America’s Choice School 
Design in 1998. Each year CPRE designs and 
conducts a series of targeted studies on the 
implementation and impacts of the America’s 
Choice design. The report presented here is one 
of this year’s evaluation reports. 

The purpose of CPRE’s evaluation is to provide 
formative feedback to NCEE and America’s 
Choice schools about emerging trends in the 
implementation of the design, and to seek 
evidence of the impacts of the design using 
accepted high standards of evaluation design and 
analysis methodologies. 

CPRE’s evaluation of America’s Choice is 
guided by three overarching evaluation 
questions about the implementation and impact 
of the design. First, is America’s Choice being 
carried out in the manner envisioned — that is, 
how are teachers and school administrators 
understanding and implementing the many 
facets of the America’s Choice reform design? 
Second, as a result of their implementation of 
America’s Choice, are the instructional practices 
of teachers changing in ways that would 
improve student learning? Third, to what degree 
can improvements in student achievement be 
attributed to the design? Within this framework, 
annual evaluation studies target specific aspects 
of the America’s Choice design for more in- 
depth investigation. 

To address these questions, the CPRE evaluation 
team gathers a broad array of qualitative and 



quantitative data to develop a rich and valid 
picture of the implementation process over time 
and to capture the impacts of the design on 
students and teachers. Our data sources include: 

• Surveys of teachers and administrators in 
America’s Choice schools nationwide. 

• Site visits to schools across the country to 
observe classroom instruction, examine 
implementation artifacts, and interview 
teachers, students, and school 
administrators. 

• Telephone interviews with NCEE staff, 
school faculty members, and school and 
district administrators. 

• Document reviews. 

• Observations of national, regional, and 
school-level professional development. 

• Collection of a variety of student 
performance measures, including state and 
local tests, the New Standards Reference 
Examination, and more authentic samples of 
student work products. 

After data collection, CPRE research team 
members analyze the data using appropriate 
qualitative and quantitative research techniques 
in order to identify patterns of intended and 
unintended consequences and to detect effects of 
the design on students, teachers, and schools. 

The results are reported in a series of thematic 
evaluation reports that are released each year. 

To inquire about the evaluation reports that are 
available, please contact CPRE’s 
communications office at cpre@gse.upenn.edu, 
visit our web site at www.cpre.org, or call us at 
(215)573-0700. 
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Executive Summary 

T his study, conducted by the 

Consortium for Policy Research in 
Education (CPRE), reports on the 
impact of the America’s Choice school 
design on student standardized test performance 
in Duval County, Florida. We present the results 
of the first two years of the impact of America’s 
Choice in Duval County in elementary and 
middle schools in writing, reading, and 
mathematics using test data from 1999-2001 . 

To detect the effects of America’s Choice on 
student learning, we compared the gains in 
performance of students in America’s Choice 
schools to those of students in other schools in 
the district. We employed two statistical 
techniques in order to make the comparisons as 
fair as possible. First, we controlled for a variety 
of student and school demographic 
characteristics, including prior student 
achievement, in order to isolate the influences of 
America’s Choice on student learning during a 
one-to-two year period. Second, we used a 
statistical method called multi-level modeling 
that allowed us to appropriately model the fact 
that students are nested within schools and to 
take into account the fact that we were looking 
for the effect of a school-level reform effort 
using individual-level student data. 

The major findings are: 

• Overall, the patterns in student standardized 
test performance indicate that students in 
America’s Choice schools regularly 
outgained students in other district schools 
in writing, and to a lesser extent reading and 
mathematics, after controlling for prior 
student achievement, and student and school 
demographic characteristics. In all three 
subjects, in both elementary and middle 
schools, there were multiple examples of 
significantly higher learning gains of 
students in America’s Choice schools in 
comparison to students in other schools in 
the district, while there were no cases where 
students in the other district schools 
statistically outgained students in America’s 



Choice schools. In many cases, the 
differences in learning were positive in favor 
of students in America’s Choice schools, but 
only approaching statistical significance, in 
part due to the small number of America’s 
Choice schools in some parts of the study. 

By contrast, there were no cases where 
students in other district schools outgained 
students in America’s Choice schools that 
were even approaching statistical 
significance. In fact, the only school-level 
variable that came up significant more often 
than America’s Choice was the percentage 
of students in a school receiving free or 
reduced-price lunch — an indicator of 
poverty that has long been documented to be 
strongly associated with student 
performance. 

• Detectable patterns of effect were strongest 
in writing, where students in America’s 
Choice schools consistently demonstrated 
higher learning gains than students in other 
schools in the district. The results were most 
dramatic in fourth grade, where results were 
statistically significant in both cohorts of 
America’s Choice schools. In eighth grade, 
students in America’s Choice schools had 
higher test performance learning gains than 
students in other district schools. These 
results were promising, although not 
statistically significant in part due to the 
small number of middle schools that were 
implementing America’s Choice in both 
cohorts II (four schools) and III (seven 
schools). The most dramatic and positive 
differences in writing in favor of America’s 
Choice were visible in the first year of 
implementation for both cohorts, with 
smaller positive differences continuing in 
the second year of implementation. 

• In reading, there were few detectable 
differences between the performance of 
students in America’s Choice schools and 
those in other schools within the district. 
There were, however, a couple of grades 
where America’s Choice students outgained 
their peers in other schools within the 
district. The largest effects in reading 
performance gains associated with 
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America’s Choice were for eighth-grade 
students in cohort II schools, where eighth- 
grade students in America’s Choice schools 
gained about 4% more than their peers in the 
other middle schools in the district. Given 
that cohort II middle schools have been 
implementing the America’s Choice reading 
program the longest, this is promising. 
Cohort III fourth graders in America’s 
Choice schools also significantly 
outperformed their peers. In no grade levels 
did students in other district schools 
statistically outperform students in 
America’s Choice schools in reading. 

• In mathematics, students in their second 
year of attending an America’s Choice 
school (cohort II) significantly outperformed 
their peers in both fourth and eighth grade. 

In fourth grade, the results were particularly 
driven by the performance of the students in 
the larger America’s Choice schools. In 
eighth grade, the students in the America’s 
Choice cohort II middle schools had 
statistically larger learning gains in 
comparison to the students in the district’s 
other schools by almost 10% — the widest 
margins for any of the analyses done for this 
study. 

• Overall, across grades and subjects, the 
America’s Choice design had an equal effect 
on White and minority students. In 22% of 
the grades and subjects examined, 

America’s Choice was associated with a 



statistically significant reduction in the gap 
in performance between White and minority 
students by increasing the performance of 
minority students at a higher rate than White 
students. The reductions in performance 
gaps were particularly persistent in eighth- 
grade cohort II America’s Choice schools in 
writing and reading. In no case did 
America’s Choice significantly exacerbate 
the gap in performance between White and 
minority students. Although in most cases 
the differences were not statistically 
significant, in 80% of the grades and 
subjects examined, America’s Choice 
schools had a reduction in the gap in 
performance between White and minority 
students. 

Finally, it is important to understand the context 
within which America’s Choice is being 
implemented in Duval County. The district 
leaders’ initiatives contain many elements that 
are consistent with America’s Choice. 

Therefore, we must consider the possibility that 
the district’s efforts to implement standards- 
based reform in all schools across the district 
may be minimizing the effects of America’s 
Choice. In other words, if other schools in the 
district are gaining in performance as a result of 
the district-wide reform efforts as well, then the 
results of America’s Choice may be less 
detectable. CPRE is designing a future study that 
will allow us to test this hypothesis. 
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Introduction 

T he America’s Choice comprehensive 
school reform design was first 
implemented in Duval County in the 
1999-2000 school year when 10 
elementary schools and four middle schools, 
together called cohort II, adopted the design. In 
the 2000-2001 school year, an additional 38 
elementary and seven middle schools, 
collectively called cohort III, became America’s 
Choice schools.’ In this study we compare the 
writing, reading, and mathematics test 
performance of the students in these schools to 
the performance of students in the other 53 
district elementary schools and 16 district 
middle schools. Other schools within the district 
were used as a comparison group, even though 
they are also adopting elements of standards- 
based reform as part of the district’s broader 
reform efforts. 

It is important to understand the rollout of the 
America’s Choice design in Duval County 
because implementation influences impact. The 
rollout of America’s Choice, particularly with 
regard to reading and writing, differed slightly 
between the two cohorts. In their first year, the 
cohort II elementary schools implemented the 
America’s Choice writing component, and the 
cohort II middle schools implemented the 
America’s Choice reading component. In their 
second year, the cohort II elementary schools 
focused on implementing the reading component 
of the design, while the middle schools focused 
on the writing component of America’s Choice. 
Both the elementary and middle schools in 
cohort III implemented writing in their first year 
and reading in their second year. Mathematics 
was initially implemented in schools’ second 
year of the design, but the pace of mathematics 
implementation varied widely by school. 

While America’s Choice was being 
implemented in 59 of the 128 elementary and 
middle schools in Duval County in 2001, other 



’ These cohort designations refer to the national 
America’s Choice cohorts with which these schools 
are members. There are no cohort I America’s Choice 
schools in Duval County. 



instructional improvement efforts were 
underway in other district schools as well. 
Eighteen elementary schools were implementing 
other school reform models like Direct 
Instruction and Success for All. Additionally, all 
other schools in the district were being 
introduced to standards and selected elements of 
America’s Choice. For example, all schools 
were given the New Standards Performance 
Standards and told that these, in addition to the 
Florida Sunshine Standards, were the focal point 
for student outcomes. The district also had an 
additional contract with NCEE to provide 
literacy institutes at all levels, where schools 
were asked to send one instructor or a small 
team of teachers to training and to implement at 
least one model literacy classroom in their 
school. Also, at the encouragement of the 
regional superintendents in Duval County, 
America’s Choice schools opened their doors to 
faculty members of other schools, which sent 
teams of teachers and administrators to 
investigate what was going on. Because of the 
potential “leakage effect” to other district 
schools, the comparisons in this study can best 
be seen as a comparison between formal 
implementation of America’s Choice and 
schools implementing some other elements of 
the design. 

This report contains five sections. Following this 
introduction, we provide a comparison of the 
descriptive statistics of America’s Choice 
schools and other district schools. Differences in 
these simple descriptive comparisons lay the 
groundwork for applying more sophisticated 
methods of analysis to the data that provide 
fairer comparisons in order to isolate the 
influence of America’s Choice on student 
performance. We then describe the methods for 
these more sophisticated analyses. Next, we 
detail the results of our analyses of the effects of 
America’s Choice on student performance in 
writing, reading, and mathematics. We also 
include an analysis of the influence of America’s 
Choice on the gaps in performance between 
minority and White students. The report 
concludes with a discussion of the findings. 
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Table 1 . Descriptive Statistics (Means and Standard Deviations) for America's 
Choice Cohort II and III Elementary Schools and Other Elementary Schools in 
Duval County in 1999, Prior to the Adoption of America's Choice 




America's Choice 


America's Choice 


Other District 




Cohort II Elementary 


Cohort III Elementary 


Elementary 




Schools 


Schools 


Schools 




(n=10) 


GO 

CO 

1) 

_c_ 


(n=53) 


Percent of students on 


68.57 


57.67 


61.40 


free/reduced-prlce lunch 


(23.93) 


(18.27) 


(23.48) 


Average class size 


22.22 


23.44 


23.13 




(2.64) 


(2.59) 


(4.75) 


Average school size 


558.50 


652.26 


643.08 




(201.18) 


(247.04) 


(285.66) 


Percent of students absent 2 1 


9.96 


9.92 


8.84 


or more days 


(3.09) 


(3.42) 


(3.67) 


Fourth-grade criterion- 


287.95 


295.86 


294.28 


referenced reading 
performance 


(54.94) 


(51.89) 


(54.41) 


Fourth-grade norm-referenced 


712.25 


721.37 


720.04 


mathematics performance 


(55.87) 


(52.21) 


(53.84) 



Descriptive 
Comparison of 
Americans Choice and 
Other District Schools 

In this section, we compare demographic data on 
both America’s Choice and other district 
schools. Overall, there were substantial 
differences in the composition of the students in 
the two cohorts of America’s Choice schools in 
comparison to the district’s other schools. The 
cohort II America’s Choice schools generally 
appeared to have poorer and lower performing 
students in comparison to other schools in the 
district, while the cohort III schools appeared to 
be more similar to other district schools. Table 1 
provides a detailed comparison between the two 
cohorts of America’s Choice elementary schools 
and other elementary schools in the district. The 
data presented in this table are from 1998-1999, 



the year before any school in the district adopted 
America’s Choice. 

The 10 America’s Choice cohort II elementary 
schools were among the poorest and lowest 
performing in the district prior to their 
implementation of America’s Choice. As seen in 
Table 1, the cohort II elementary schools had a 
higher proportion of students receiving free or 
reduced-price lunch (69% compared to 61%), 
and were slightly smaller, on average, than other 
schools in the district. As shown in the last two 
rows of Table 1, prior to their adoption of 
America’s Choice, the cohort II elementary 
schools had substantially lower reading and 
mathematics test scores in comparison to other 
district elementary schools. In fact, the cohort II 
school average test performance was 
significantly lower (reading: t=3.07, d.f =5659, 
p<001; mathematics: t=3.85, d.f.= 5794, 
p<.001) than it was for other elementary schools 
in the district. 
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Table 2. Descriptive Statistics (Means and Standard Deviations) for America's 
Choice Cohort II and III Middle Schools and Other Middle Schools in Duval County 
in 1999, Prior to the Adoption of America's Choice 




America's Choice 
Cohort II 
Middle Schools 
(n=4) 


America's Choice 
Cohort III 
Middle Schools 
(n=7) 


Other 

District 

Middle Schools 
(n=16) 


Percent of students on 


67.40 


40.37 


54.68 


free/ reduced-price lunch 


(17.59) 


(14.52) 


(22.54) 


Average class size 


24.58 


26.99 


25.05 




(4.60) 


(2.52) 


(4.21) 


Average school size 


1,153.25 


1,336.00 


967.88 




(319.88) 


(200.74) 


(453.96) 


Percent of students absent 2 1 or 


32.20 


14.96 


16.61 


more days 


(5.23) 


(7.77) 


(12.32) 


Eighth-grade criterion-referenced 


272.67 


299.09 


294.61 


reading performance 


(46.37) 


(48.10) 


(51.55) 


Eighth-grade norm-referenced 


268.23 


299.52 


291.00 


mathematics performance 


(52.23) 


(47.54) 


(54.54) 



The second group of 38 America’s Choice 
elementary schools that adopted the design in 
2000 were more demographically similar to the 
other 53 elementary schools in the district. They 
had similar percentages of students receiving 
free or reduced-price lunch (58% compared to 
61%), and were of similar size (652 students, on 
average, compared to 643 students for other 
district elementary schools). In 1999, prior to 
their adoption of America’s Choice, the reading 
and mathematics test scores in cohort III 
elementary schools were also similar, on 
average, to those of other district elementary 
schools. 

Middle school comparisons between America’s 
Choice schools and other middle schools in 
Duval County showed a similar pattern to that of 
elementary schools. Overall, the cohort II 
schools served poorer students who were lower 
performing than their peers in other district 
middle schools, while the cohort III schools 
were more similar to the other district middle 
schools. Table 2 shows the descriptive statistics 
for the middle schools in cohorts II and III of 
America’s Choice as well as the other 16 middle 
schools in Duval Coimty. 



There were only four middle schools in cohort II 
of America’s Choice in Duval County. In 
comparison to the other middle schools in the 
district, these four schools had a higher 
percentage of students receiving free or reduced- 
price lunch (67% compared to 55%). These were 
also larger schools, serving an average of 1,153 
students, compared to the average of 968 for 
other district middle schools. They also had a 
serious student attendance problem: almost a 
third of their students had high absence rates, 
compared to 17% in the district’s other middle 
schools. Prior to their adoption of America’s 
Choice, in 1999, the cohort II schools had 
significantly lower test performance than that of 
the other district middle schools (reading: 
t=14.07, d.f.=2131, p<.001; mathematics: 
t=12.85, d.f =5290, p<.001). In reading, this 
represented almost a half a standard deviation 
lower performance. In other words, in the 
America’s Choice middle schools, 
approximately 65% of the students scored below 
the district average of 293, while in other district 
middle schools, only 44% scored below the 
district average. 
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The seven cohort III middle schools were more 
similar to the district’s other middle schools, 
although they had a lower percentage of students 
receiving free or reduced-price lunch (40% 
compared to 55%) and were larger, on average, 
than the district’s other middle schools (1,336 
students compared to 968 students). In terms of 
achievement prior to involvement in America’s 
Choice, the cohort III schools had higher 
performance in both reading (299 compared to 
295) and mathematics (300 compared to 291) 
compared to other district middle schools. Both 
of these differences were statistically significant 
(reading: t=3.61, d.f.=5773, p<.001; 
mathematics: t=6.75, d.f.=6091, p<.001). 

Overall, there were many differences in the 
demographics and test performance of the 
students in the two cohorts of America’s Choice 
schools in comparison to the district’s other 
schools. Because of the differences between the 
two cohorts of America’s Choice schools and 
the other district schools, both in terms of the 
demographics and prior achievement of their 
students, it is important to adjust for these 
differences using statistical means. Statistical 
controls allow us to equalize differences 
between two groups, helping to produce fairer 
comparisons. 

Methodology 

In this section, we describe the data we 
examined in our analyses, including test scores 
and student and school demographic 
information. We also detail the methods that we 
used to analyze the effects of America’s Choice 
on students’ test score gains after controlling for 
the differences in the backgrounds of the 
students and demographics of the schools. 

Data 

In conducting these analyses, we used data from 
about 23,000 fourth and fifth graders enrolled in 
101 elementary schools, and 20,000 seventh and 
eighth graders from 27 middle schools in Duval 
County. These students constitute the full 
population of students that attended Duval 
County schools at these grade levels between 
1999 and 2001. Student demographics and 



district and state standardized test scores for 
students in third through eighth grade were 
collected for the 1998-1999, 1999-2000, and 
2000-2001 school years. The standardized tests 
were administered in the spring of each school 
year. 

Our analyses consisted of a series of models that 
controlled for each students’ achievement in the 
previous year, as well as student and school 
demographic characteristics, in order to predict 
the current year’s achievement. Thus, for 
example, we predicted fourth-grade achievement 
controlling for third-grade achievement. We 
typically did analyses that predicted the 
achievement of fourth and fifth graders in 
elementary schools, and sixth, seventh, and 
eighth graders in middle schools. The 1998-1999 
school year served as the baseline (i.e., prior to 
their adoption of America’s Choice) year for the 
cohort II analysis, and the 1999-2000 school 
year served as the baseline year for the cohort III 
analysis. By the end of the 2000-2001 school 
year, the cohort II schools had implemented the 
America’s Choice design for two years and the 
America’s Choice cohort III schools for one 
year. School demographics were garnered from 
the Florida School Indicators Report for the 
1998-1999 school year. Student and school 
variables are described below. 

We considered two general methods for 
constructing appropriate comparison groups for 
the two cohorts of America’s Choice schools. 

We first considered drawing a matched 
comparison group for each of the America’s 
Choice cohorts. There were two problems with 
this approach. First, since the America’s Choice 
schools were some of the lowest performing 
schools in the district, it was difficult to find 
matched comparisons for many of them. Second, 
the strategy of matched comparisons reduced 
our sample sizes such that we had less likelihood 
of detecting differences in the test performance 
of the students in the two groups of schools. 
Therefore, we decided to use the entire 
population of schools in the district, relying on 
statistical techniques to adjust for differences in 
student and school demographic characteristics. 

Cases with missing data were removed by 
listwise deletion. Almost all students who were 




4 



12 



Impact of America's Choice on Student Performance 



Supovitz, Taylor, and May 



Table 3. Test Data Used for Analyses of the Impact of America's Choice on 
Student Performance in Duval County from 1999 to 2001 

Grade Level 

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 



Reading 



2001 


SAT-9 


SSS 


SAT-9 


SAT-9 


SAT-9 


SSS 


2000 


SAT-9 


SSS 


SAT-9 


SAT-9 


SAT-9 


SSS 


1999 


ctbsant 


SSS 


ctbsant 


CTBS/TNT 


ctbsant 


SSS 


Writing 














2001 




SSS 








SSS 


2000 




SSS 








SSS 


1999 




SSS 








SSS 


Math 

2001 


SAT-9 


SAT-9 


SSS 


SAT-9 


SAT-9 


SSS 


2000 


SAT-9 


SAT-9 


SSS 


SAT-9 


SAT-9 


SSS 


1999 


ctbsant 


CTBS/TNT 


SSS 


CTBS/TNT 


CTBS/TNT 


SSS 



SAT-9 = Stanford Achievement Test, Version 9, a norm-referenced assessment 

SSS = Sunshine State Standards test, a criterion-referenced assessment 

CTBSANT = California Test of Basic Skills/TerraNova Test, norm-referenced assessments 



removed from the study were omitted either due 
to missing pre-test or post-test scores. Overall, 
22-30% of elementary school students and 28- 
38% of middle school students were excluded 
from the analyses, depending on the grade and 
year. These high missing data rates were largely 
due to student mobility. Patterns within the 
missing data were explored for potential biases 
in the missing data of students in America’s 
Choice schools and those in other district 
schools. Comparisons of the available pre- or 
post-test scores for those removed did not 
indicate any substantial differences in the 
proportion of students removed from either of 
the comparison groups of interest. 

Test Scores 

Florida uses a mixture of norm-referenced and 
criterion-referenced assessment measures to 
assess student performance at different grade 
levels. Norm-referenced assessments depict the 
performance of students in relation to other 
students, while criterion-referenced assessments 
are intended to determine how well a student has 
mastered a set of standards representing an 
entire curriculum. In grades where students took 



a norm-referenced and a criterion-referenced 
assessment, we chose to use the criterion- 
referenced measure in our analysis because it is 
closer to the standards-based philosophy of 
America’s Choice. 

Table 3 shows the test data used in CPRE’s 
analyses for each grade level, subject, and year. 
The norm-referenced assessments at grade three 
were only used to control for the prior 
achievement of fourth graders. The fourth-grade 
analyses used writing and reading scores on the 
criterion-referenced Sunshine State Standards 
(SSS) portion of the Florida Comprehensive 
Assessment Test (FCAT), and mathematics 
scores on the SAT-9, the norm-referenced 
portion of the FCAT. The fifth-grade analyses 
used mathematics scores on the criterion- 
referenced portion of the FCAT, and reading and 
mathematics scores on the norm-referenced 
portion of the FCAT. The sixth- and seventh- 
grade analyses used reading and mathematics 
scores on the norm-referenced portion of the 
FCAT. Eighth-grade analyses used reading, 
writing, and mathematics scores on the criterion- 
referenced portion of the FCAT, and reading and 
mathematics scores on the norm-referenced 
portion of the FCAT. 
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Our basic analytic strategy in using these test 
score data was to examine student gain scores by 
predicting a particular year’s student 
performance and controlling for the students’ 
prior year test score. Therefore, our models at 
times controlled for a norm-referenced 
assessment and predicted a criterion-referenced 
assessment. Achievement scores were 
transformed into z-score units for the analyses so 
that we could compare the magnitude of effects, 
regardless of the test and subject area. 

Student and School 
Demographic Characteristics 

Six individual student variables and six school- 
level variables were used as predictors in the 
models. Student predictor variables included 
prior standardized achievement score; number of 
days absent in the current school year; and 
dummy indicators for free or reduced-price 
lunch, minority student, male student, and 
disability classification. School-level predictor 
variables included school size, average class 
size, school grade (A-F) assigned under the state 
accountability system,^ percentage of students 
eligible for free or reduced-price lunch, 
percentage of students absent 21 or more days, 
and a dummy variable for schools that 
implemented the America’s Choice design. 

Many of the descriptive statistics for these 
variables were provided in Tables 1 and 2. 

Models 

Multi-level models were used to estimate the 
effects of America’s Choice on student 
achievement in reading, writing, and 
mathematics after adjusting for selected student 
and school characteristics. Models were 
estimated separately for each cohort and grade 
level. Each model comprised two levels where 
individual students (Level 1) were nested within 
schools (Level 2). The basic random intercept 
model equation follows. 



^ The school grades were converted to a numerical 
scale so that A=l, B=2, and so on. 
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Student-level model (Level 1): 

Yij = poj + Pi(PRIOR ACHIEVEMENT)ij + 
P2(LUNCH ASSISTANCE)ij + P3(MINORITY)ij 
+ P4(MALE)ij + P5(DISABILITY)ij + 
P6(ABSENCES)ij + ry 

School-level model (Level 2): 

Poj = Yoo Yoi(SCHOOL GRADE)j + 

Yo 2(LUNCH ASSISTANCE)j + yo3(SCHOOL 
SIZE)j +Yo 4(CLASS SIZE)j + Yos(ABSENT 21+ 
Daysjj + Yo6( AMERICA’S CHOICE)j + Uoj 

In the Level 1 model, Yij is the student 
achievement outcome for student i in school j, 

Poj is the mean student achievement in school j, 
Pi through Pe are the fixed effects coefficients 
for the student-level predictors, and ry is the 
individual student-level random effect, or error 
term. In the Level 2 model, Yoo is the mean 
student achievement gain (i.e., the residualized 
achievement gain after controlling for prior 
achievement) across all schools. Yoi through Yos 
are the fixed effects coefficients for the school- 
level predictors after controlling for the Level 1 
student characteristics, Yo 6 is the effect of 
America’s Choice on school mean achievement 
gain after controlling for the student and other 
school-level characteristics, and Uoj is the school- 
level random effect, or error term. Student-level 
sample sizes varied in the analyses due to 
listwise deletion of cases with missing 
demographics or test data. 

In follow-up analyses, interactions between the 
America’s Choice predictor and each of the 
school-level predictors were also explored. The 
minority achievement gap was also explored by 
adding a random slope to the above two level 
models. 

p3j = Y30 + Y3i(AMERICA’S CHOICE)j + U3j 

Where Y30 is the average minority gap after 
controlling for prior achievement score, lunch 
assistance, gender, disability, and absences; Y31 is 
the effect of America’s Choice on the minority 
achievement gap after controlling for the five 
student characteristics; and U 3 j is the school-level 
random slope effect. 
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Table 4. Differences in Standardized Units, by Grade and Cohort, Between the 
Writing Performance of Students in America's Choice Schools and Students in 
Other District Schools (with Standard Errors in Parentheses) 


Cohort and Grade 


1999-2000 


2000-2001 


1999-2001 


Cohort II 








Fourth Grade ~ ^ ^1 


.18 


.14 






(.09) 


(.10) 




Eighth Grade = 4) 


.07 


.09 


.09 




(.14) 


(.13) 


(.12) 


Cohort III 








Fourth Grade = 38) 




.14* 








(.06) 




Eighth Grade = 7) 




.10 








(.10) 





p < .10, * p < .05, ** p < .01 



Results 

In this section, we report the findings of our 
analyses comparing the performance of students 
in America’s Choice schools to that of students 
in other district schools in three subjects: 
writing, reading, and mathematics. We also 
report on a series of analyses that looked at the 
relative performance of White and minority 
students to investigate whether school-level 
differences in the performance gains of students 
of different ethnicities was associated with 
attending an America’s Choice school. 

Impact' of America's Choice 
on Student Writing 
Performance 

Students in America’s Choice schools 
consistently outperformed students in other 
district schools in writing after controlling for 
differences in students’ prior achievement and 
differences in the demographic characteristics of 
students and schools. Table 4 shows the results, 
by grade and cohort, of a series of multi-level 
models that compared the performance of 
students on the FCAT criterion-referenced 
assessment in writing after controlling for 
student demographic characteristics, student 
prior achievement (as measured by third- and 



seventh-grade norm-referenced reading scores), 
and school demographic characteristics.^ 

Writing assessments are only administered in 
fourth and eighth grades in Duval County. 

For cohort II, which had been implementing 
America’s Choice for two years when these data 
were made available, the results are depicted in 
three ways: the first column shows the effects 
after the first year of implementation, 1999- 
2000; the second column shows the effects after 
the second year of implementation, 2000-2001, 
examining a different group of students; and the 
third column shows the effects over two years of 
implementation, 1999-2001, for the same group 
of students. Thus, using eighth graders as an 
example, the first column shows the results for 
eighth graders in 1999-2000, controlling for 
their seventh-grade test scores. The second 
column shows the results for the 2000-2001 
eighth graders. The third column shows the 
results for the 2000-2001 eighth graders (the 
same students as in the second column) 
controlling for their sixth-grade test scores (i.e., 
performance over two years). The reason that 
there are no 1999-2001 analyses for fourth 
graders is because we would need a second- 
grade test as a control for prior achievement and 
students are not assessed until the end of the 



^ See Appendix A for a full set of the multi-level 
model results for writing. 
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third grade. For cohort III, which had been 
implementing America’s Choice for just one 
year when these data were made available, the 
results are shown just for the 2000-2001 school 
year. 

All the numbers represented in Table 4, as well 
as all following tables, show the differences, in 
standard deviation units, between students in 
America’s Choice schools and those in other 
Duval County schools, for different subjects and 
grade levels, after controlling for both individual 
student characteristics and school characteristics. 
Reporting the scores in standardized units is 
commonly accepted as it allows for comparisons 
of effect sizes across years and subjects. 

After the first year of implementation of 
America’s Choice (1999-2000), the fourth-grade 
students in the 10 America’s Choice cohort II 
schools statistically significantly outgained 
fourth-grade students in the 53 other district 
elementary schools by about a fifth of a standard 
deviation. In non-technical terms, think of two 
students who performed similarly in writing in 
the year prior to the adoption of America’s 
Choice. They both attended similar schools, but 
one was an America’s Choice school and the 
other was not. At the end of the year, the student 
in the America’s Choice school gained about 5% 
more on the test than their peer in the other 
district elementary school."^ 

In 2000-2001, on average, the fourth-grade 
students in the cohort II America’s Choice 
schools again outgained those in the other 
district schools, although this difference was 
only approaching statistical significance. For 
eighth-grade cohort II writing performance, the 
students in the America’s Choice schools had 
test score gains that were, on average, larger 
than their peers in the other district middle 
schools, as represented by the positive 
coefficients, but these differences were not 
statistically significant. Thus, the differences 
between the two groups of students may be due 
to chance. This may be due to the fact that there 
are only four middle schools in cohort II of 



This amounted to a 1.5-point difference on the 2000 
Sunshine State Standards Writing Test where 90% of 
the students scored between 1 5 and 45 . 



America’s Choice, which decreases the 
precision of the estimates of difference. 

The results for cohort III America’s Choice 
schools show a similar pattern. After their first 
year in America’s Choice schools, fourth-grade 
students had statistically significantly higher 
performance gains in writing than did students in 
other standards-based schools in the district. 
After controlling for student background 
characteristics and prior achievement, fourth- 
grade students in cohort III America’s Choice 
schools outperformed fourth graders in other 
district schools by . 14 of a standard deviation. 
This represents about a 4% difference in 
students’ learning gains in writing that are 
associated with America’s Choice, as measured 
by the Sunshine State Standard writing test.^ 

Figure 1 • Predicted Writing 
Performance of Cohort III Fourth- 
grade Students in America's Choice 
and Other District Schools 




^ This amounted to a 1 .25-point difference in test 
score gains, where 90% of the students scored 
between 20 and 50 points. 
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This result is shown graphically in Figure 1, 
which portrays the predicted performance of 
fourth-grade students in America’s Choice and 
other district schools. The regression line shows 
that the predicted gains of the other district 
schools is .14 of a standard deviation below the 
predicted gains of students in America’s Choice 
schools. In other words, students in America’s 
Choice schools were predicted to gain .09 of a 
standard deviation more than the average gain of 
students in the district. By contrast, the students 
in the other district elementary schools were 



predicted to gain .05 of a standard deviation less 
than the average gain in the district. 

Finally, the eighth-grade students in the seven 
cohort III America’s Choice schools performed 
a tenth of a standard deviation higher, on 
average, than did students in the other 16 middle 
schools in the district after controlling for prior 
achievement and individual student and school 
demographic characteristics. While this 
difference was promising, we cannot rule out the 
possibility that this difference was due to 
chance. 



Table 5. Differences, by Grade and Cohort, Between the Reading Performance of 
Students in America's Choice Schools and Students in Other District Schools (with 
Standard Errors in Parentheses) 



Cohort and Grade 


1999-2000 


2000-2001 


1999-2001 


Cohort II 

Fourth Grade = 10) 


-.04 


.02 






(.06) 


(.04) 




Fifth Grade ^ ^1 


-.04 


-.01 


-.02 


(.05) 


(.04) 


(.05) 


Sixth Grade = 4) 


.02 


.02 


.00 


(.04) 


(.06) 


(.06) 


Seventh Grade (n^^~ = 4) 


-.03 


-.04 


-.01 


(.04) 


(.10) 


(.06) 


Eighth Grade (n^^ = ^1 


.13' 


.1 1 


.iT 


(.07) 


(.08) 


(.05) 


Cohort III 

Fourth Grade (n^^» = 38) 




.05' 








(.03) 




Fifth Grade (n^^» = 38) 




.02 






(.02) 




Sixth Grade (n^^» = 7) 




.01 






(.04) 




Seventh Grade (n^^ = ^1 




-.08 






(.08) 




Eighth Grade (n^^ = 7) 




.02 






(.07) 





p < .1 0, * p < .05, ** p < .01 
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Impact of America's Choice 
on Student Reading 
Performance 

Overall, there were few detectable differences 
between the standardized reading performance 
of either elementary or middle school students in 
America’s Choice schools and those in the other 
schools in the district. Table 5 shows the results 
of student performance, by grade level and 
cohort, in America’s Choice schools and those 
in other standards-based schools in the district.^ 

For cohort II, as seen in Table 5, there were no 
statistically significant differences between 
students in the America’s Choice schools and 
students in the district’s other schools with the 
exception of eighth graders. Most of these 
results show, on average, slightly higher 
performance for the students in the district’s 
other schools, but the differences are so small (in 
all cases smaller than the surrounding error) that 
the performance of the two groups is virtually 
indistinguishable. 

The one exception to this pattern of negligible 
differences in reading performance between 
students in America’s Choice cohort II schools 
and students in other district schools was eighth 
grade. In both 1999-2000 and 1999-2001 (which 
represents different groups of students), eighth- 
grade students in America’s Choice schools 
statistically outperformed other district eighth 
graders in reading after controlling for prior 
reading achievement and both individual and 
school demographic characteristics. In 1999- 
2000, the gains in achievement associated with 
America’s Choice amounted to .13 of a standard 
deviation, which translates into about eight test 
score points, or 4% higher performance for 
eighth graders in America’s Choice schools after 
controlling for differences in student and school 
demographic characteristics.’ This statistically 
significant difference is particularly noteworthy 



^ See Appendices B and C for a fiill set of the multi- 
level model results for reading. 

’ These eight points on the state’s criterion- 
referenced assessment were on a test where 90% of 
the students scored between 174 and 367. 
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given the fact that there are only four cohort II 
America’s Choice middle schools, which means 
that the differences would have to be substantial 
to be statistically detectable. 

Table 5 also shows the reading results for cohort 
III students after their first year in America’s 
Choice schools, the 2000-2001 school year. 
Fourth-grade students in cohort III America’s 
Choice schools statistically significantly 
outgained fourth graders in other district 
elementary schools by one-twentieth of a 
standard deviation. These differences after one 
year, however, were fairly small, amounting to 
about a 2% difference in gains in performance 
between fourth graders in America’s Choice 
schools and fourth graders in other district 
elementary schools.® The results in the fifth, 
sixth, seventh, and eighth grades indicated that 
there were no detectable differences in reading 
performance gains on the standardized tests 
between students in the cohort III America’s 
Choice middle schools and students in the other 
district middle schools. 

Figure 2. Cohort II Eighth-grade 
Adjusted Reoding Achievement Gains 
for America's Choice and Other 
District Middle Schools from 
1999-2001 




® There was a three-point difference on a test where 
the range of scores was 1 92 for students in the fifth 
percentile and 389 for students in the 95* percentile. 
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These differences in cohort II eighth-grade 
reading performance for students in America’s 
Choice schools in comparison to students in 
other district middle schools are shown 
graphically in Figure 2. Each dot in Figure 2 
represents the adjusted school-average 
achievement gain for an individual school. The 
array of schools on the left of the figure show 
the distribution of adjusted performance gains 
for the non- America’s Choice middle schools in 
Duval County. The four dots on the right show 
the adjusted gains of the four cohort II 
America’s Choice middle schools. The solid line 
bisecting each of the two arrays represents the 



statistically significant difference in average 
performance gains between the two groups of 
schools. 

Impact of America's Choice 
on Student Mathematics 
Performance 

We sought to examine the influence of 
America’s Choice on student standardized test 
performance in mathematics in order to explore 
both direct and indirect effects. Mathematics is 



Table 6. Differences, by Grade and Cohort, Between the Mathematics 
Performance of Students in America's Choice Schools and Students in Other 
District Schools (With Standard Errors in Parentheses) 



Cohort and Grade 


1 999-2000 


2000-2001 


1999-2001 


Cohort II 

Fourth Grade ~ ^ ^1 


-.07 


.00 




Main Effect 


(.05) 


(.05) 




Fourth Grade 




.1 r 




School Size Interaction 




(.06) 




Fifth Grade (n^,- = 1 0) 


.00 


.09 


.09 




(.06) 


(.07) 


(.08) 


Sixth Grade (n^,- = 4) 


-.03 


.00 


.00 




(.05) 


(.09) 


(.08) 


Seventh Grade (n^,- = 4) 


-.06 


.02 


-.03 


(.04) 


(.14) 


(.10) 


Eighth Grade (n^,- = 4) 


.28” 


.15 


.16’ 


(.09) 


(.11) 


(.07) 


Cohort Hi 

Fourth Grade (n^^ = 38) 




.04 








(.03) 




Fifth Grade {r\^^ = 38) 




.03 






(.05) 




Sixth Grade (n^,- = 7) 




.01 






(.05) 




Seventh Grade (n^,- = 7) 




.02 

(.11) 




Eighth Grade (n^,- = 7) 




.05 






(.09) 





p < . 1 0, * p < .05, ** p < .01 
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introduced in America’s Choice schools in the 
second year of the design. Therefore, at least 
some of the teachers in cohort II America’s 
Choice schools had received mathematics 
training by NCEE and most others had received 
administered they had not yet received training 
specific to mathematics. We also sought to 
examine indirect effects of America’s Choice on 
training on delivering standards-based 
instruction at the time of the 2001 mathematics 
assessment. Teachers in cohort III schools had 
received training on delivering standards-based 
instruction, but at the time the tests were 
students’ mathematics performance because 
some researchers, as well as other CPRE 
research, have suggested that student literacy 
skills have an influence on students’ 
mathematics test performance. It makes sense 
that students’ reading and writing skills would 
influence their mathematics performance, 
particularly on today’s assessments that often 
include word problems and require higher-order 
thinking skills. 

Table 6 shows the results of our examination of 
the influence of America’s Choice on student 
performance in mathematics after controlling for 
student prior achievement and student and 
school demographic characteristics.^ For cohort 
II, there were three analyses that showed 
statistically significant differences that favored 
students in America’s Choice schools, while 
there were no cases where students in other 
district schools significantly outperformed 
students in America’s Choice schools. 

The first statistically significant result was found 
in the performance of fourth graders in 
mathematics in schools that were in their second 
year of implementation of America’s Choice 
(cohort II). But this result was not found 
consistently across all cohort II fourth-grade 
mathematics performance. Rather, there was an 
interaction in fourth-grade mathematics between 
students who were in America’s Choice schools 
and the size of their school. This indicates that 
students in large America’s Choice schools 
performed significantly better than students in 
other district schools, while students in other 



^ See Appendices C and D for a full set of the multi- 
level model results for mathematics. 
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America’s Choice schools performed no 
differently in mathematics, on average, than 
students in other district schools. These 
differences were essentially driven by the strong 
performance of the two larger than average 
America’s Choice elementary schools in cohort 
II. 

This relationship between attendance in an 
America’s Choice elementary school and school 
size is depicted graphically in Figure 3. In 
Figure 3, school-average performance gains, 
after controlling for both student and school 
demographic characteristics, are shown arrayed 
by school size. School size is represented in 
standard deviation units, with those with smaller 
than average school size on the left side of the 
figure, and those with larger than average school 
size on the right side of the figure. 

Figure 3. Fourth-grade Adjusted 
Mathematics Achievement Gains for 
America's Choice and Other District 
Elementary Schools in 2000-2001, 
Arrayed by School Size 




Several things can be seen in Figure 3. First, the 
distribution is skewed to the right, indicating 
that there are a number of particularly large 
elementary schools, virtually all of which are not 
America’s Choice schools. Second, the two 
regression lines show the predicted performance 
of the two groups of schools (America’s Choice 
and other district schools). The fact that the two 
lines are not parallel merely demonstrates that 
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the relationship between the performance of the 
two groups is not consistent at all levels of 
school size. Two particular schools were about 
one standard deviation larger than the average 
school, and their students’ mathematics 
achievement gains were larger than the average 
student achievement gains. The strong 
performance of these two schools appear to be 
driving the differences in fourth-grade 
mathematics performance between America’s 
Choice cohort II schools and other district 
elementary schools. 

The other particularly noteworthy results were 
found in the differences in mathematics 
performance between eighth graders in cohort II 
America’s Choice schools and the performance 
of eighth graders in other district middle 
schools. In 1999-2000, at the end of their first 
year of America’s Choice, there was a 
statistically significant (.28 of a standard 
deviation) and educationally substantial 
difference between the performance gains of 
eighth graders in America’s Choice schools and 
those of their peers in other district middle 
schools. This effect translates to 9% higher 
performance gains for students in America’s 
Choice schools in comparison eighth graders in 
other district middle schools.'® While this result 
can be considered an indirect effect because it 
occurred before America’s Choice provided 
training for teachers in mathematics, it is 
particularly notable because it was the largest 
effect of all the analyses done in this study. 

A result that can be argued to be more directly 
related to America’s Choice mathematics 
training can be seen for cohort II eighth-grade 
students’ performance gains beginning in 2000- 
2001, the year that teachers began receiving 
America’s Choice mathematics professional 
development. The 2000-2001 results show a 
higher, on average, gain in performance for the 
eighth graders in America’s Choice schools in 
comparison to their eighth-grade counterparts in 
other district middle schools. This difference, 
however, was not statistically significant. The 

This represents a 16-point difference on the eighth- 
grade state test for which students at the fifth 
percentile scored 1 88 points and those at the 95®' 
percentile scored 376 points. 




results for 1999-2001, however, do show a 
statistically significant higher mathematics 
performance gain for eighth-grade students in 
comparison to their peers in other district middle 
schools. There were no differences in either 
elementary or middle schools in mathematics 
performance for students in cohort III America’s 
Choice schools and students in other district 
schools after their first year in the design. 

Impact of America's Choice 
on the Differences in 
Performance Between White 
and Minority Students 



Educators have long been concerned about the 
large and persistent differences in performance 
between White and minority students. Many 
efforts have sought to make education more 
equitable by reducing these gaps in performance. 

America’s Choice sets an explicit goal of having 
all but the most severely handicapped students 
meet high standards of performance. Our final 
set of analyses explored the differential 
performance of White and minority students in 
Duval County’s America’s Choice schools. We 
sought to explore whether America’s Choice 
was influencing the gaps in performance 
between minority and White students in 
America’s Choice schools relative to the gaps in 
performance in other district schools. Our 
expectation was that, at best, America’s Choice 
would be reducing gaps in performance between 
White and minority students, and, at the very 
least, having no influence on the already existing 
differences in performance. 

To test the relative influence of America’s 
Choice on the gaps in majority/minority student 
performance, we conducted a series of analyses 
that explored interactions between student 
attendance in an America’s Choice school and 
the differences in minority AVhite student test 
performance." The analyses essentially 
compared the relative gains of minority and 



' ' “Minority” includes African American and 
Hispanic students. 
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Figure 4. Graphic Depiction of Both Within and Between Group Gaps in Student 

Performance Gains 

America’s Choice Other District Schools 
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White students in America’s Choice schools (the 
within-America’s Choice gap in performance) to 
the relative gains of minority and White students 
in other district schools (the within-non- 
America’s Choice schools gap in performance) 
and compared these two gaps. If the gap in the 
gains in America’s Choice schools was smaller 
than the gap in the gains of students in other 
district schools, then we can say that America’s 
Choice was associated with a reduction in the 
gaps in performance relative to other district 
schools (the between-school gap in 
performance). 

These relationships are described graphically in 
Figure 4 to assist the reader in understanding the 
basic underlying concepts. Within the America’s 
Choice group of schools there exists a gap in 
performance gains between White and minority 
students (the within-group gap). This same gap 
in performance gains exists between White and 
minority students in the other district schools. 
Our analyses explore whether the gaps in 
performance between the America’s Choice 
gains of students in America’s Choice schools 
are reduced relative to the gaps in performance 
if they become smaller, then the gaps in the 
performance gains of students in America’s 
Choice schools are reduced relative to the gaps 
in performance gains of students in the other 
district schools. If they become larger, then the 
gaps in the performance gains of students in 
America’s Choice schools are exacerbated 
relative to the gaps in performance gains of 
students in other district schools. 



Table 7 provides a summary of the results of our 
analyses. The full set of numbers are provided in 
appendices E, F, and G. In Table 7, a minus sign 
(-) indicates that the gaps in the performance 
gains of students in America’s Choice schools 
are reduced relative to the gaps in performance 
gains of students in the other district schools. A 
plus sign (+) indicates that the gaps in the 
performance gains of students in America’s 
Choice schools are increased relative to the gaps 
in performance gains of students in other district 
schools. 

Several patterns are evident from Table 7. The 
overall pattern indicates that in most cases there 
were no differences between the performance 
gaps in America’s Choice schools and the 
performance gaps of other district schools. Ten 
of the 45 analyses, or 22%, were statistically 
significant. All of these statistically significant 
results show that the gaps in the performance 
gains were significantly smaller in America’s 
Choice schools in comparison to other district 
schools. There were no cases where America’s 
Choice significantly increased the gaps in 
performance in comparison to other district 
schools. Further, except for one other case, all of 
the reductions in the gaps in performance gains 
that were statistically significant occurred in the 
eighth grade of cohort II schools in reading and 
writing. For that group of eighth graders, 
America’s Choice appeared to have a persistent 
and positive influence on reducing the gaps in 
performance between minority and majority 
students. 
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Table 7* Impact of Americans Choice on the Gaps in Performance Gains Between 
White and Minority Students in Duval County^ 1999-2001 

Writing Reading Mathematics 

Cohort and Grade 1999- 2000- 1999- 1999- 2000- 1999-1999- 2000- 1999- 

2000 2001 2001 2000 2001 2001 2000 2001 2001 

Cohort II 



Fourth Grade (n=10) 

Fifth Grade (n=10) 

Sixth Grade (n=10) 

Seventh Grade (n=4) 

Eighth Grade (n=4) 

Cohort III 

Fourth Grade (n=38) 

Fifth Grade (n=38) 

Sixth Grade (n=10) 

Seventh Grade (n=7) 

Eighth Grade (n=7) 
p < .1 0, * p < .05, *’ p < .01 



+ 



+ + 



+ 



+ 



+ 



+ 



Summary 

The results from this study indicate that 
America’s Choice is having a positive influence 
on student standardized test performance, most 
notably in writing and to a lesser extent in 
reading and mathematics. In writing, in both 
elementary and middle schools, the performance 
gains of America’s Choice students were 
persistently higher than they were for similar 
students in other district schools. In reading, 
there were some examples of differences, but the 
patterns were erratic. The most distinct effects in 
favor of America’s Choice in reading were 
eighth grade in cohort II and fourth grade in 
cohort III. In mathematics, fourth and eighth 
graders in cohort II America’s Choice schools 
significantly outgained their counterparts in 
other district schools. Thus, while the patterns of 
statistically significant effects favoring 
America’s Choice were not overwhelming in 
reading and mathematics, there was not a single 



counterfactual case where students in other 
district schools significantly outgained students 
in America’s Choice schools. 

What might explain why there were stronger 
patterns of difference in student writing 
performance, but less so in reading? One 
possible explanation may be found in the 
America’s Choice rollout strategy for reading. 
The design calls for elementary schools to 
implement reading in their second year, while 
middle schools implement reading in their first 
year. This explains, in part, the pattern of effect 
for cohort II reading performance, where eighth- 
grade students in America’s Choice schools 
significantly outperformed their peers in other 
district schools, while there were no differences 
in the performance of America’s Choice 
elementary students in comparison to students in 
other district elementary schools. This argument 
is undermined somewhat by the lack of 
differences in performance in the seventh-grade 
cohort II schools. 
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Another plausible hypothesis, well documented 
in the literature on teaching reading, is that 
reading instruction is more difficult for teachers 
to master than writing instruction, and therefore 
takes longer to implement. While writing 
produces actual products that teachers can 
examine at any time, reading is a more 
individualized activity, occurring in real time. 
Effective teachers must know multiple 
developmental strategies and understand how to 
monitor students more carefully to develop their 
reading skills. If it takes longer for teachers to 
acquire the expertise to deliver effective reading 
instruction, then it follows that the effects on 
student reading performance would take longer 
to manifest themselves in student test 
performance. Finally, we cannot ignore the 
possibility that the America’s Choice 
instructional approach to reading is less effective 
than its writing counterpart. 

While this study contributes to the knowledge 
base of the influence of the America’s Choice 
program on student performance, many 
questions remain to be answered in upcoming 
CPRE studies both in Duval County and other 
districts across the United States. First, we seek 
to duplicate the Duval analyses in other districts 
across the country to seek replicable evidence of 
the impact of America’s Choice on student 
learning. 

Second, in Duval County we intend to explore 
whether the effects of America’s Choice on 
student performance are being minimized by the 
district’s system- wide standards-based reform 
efforts. Some advocates of America’s Choice 
hypothesize that the implementation of elements 
of the America’s Choice design in other schools 
in Duval County through the district’s standards- 
based reform efforts are reducing the differences 
between the performance of America’s Choice 
schools and other district schools because other 
district schools are also improving in 
performance. CPRE plans to explore this 
hypothesis by conducting an analysis of gains in 
student performance using another Florida 
district that is demographically similar to Duval 
County as a comparison site. 



Third, we plan to investigate the influences of 
implementation on schools’ gains in student test 
performance. CPRE’s national study of the 
implementation of readers and writers workshop 
found that implementation is variable both 
within and between schools. It seems logical that 
differential implementation rates would 
influence school-average gains in student 
learning, and CPRE plans to contribute 
empirical evidence to test the relationship 
between implementation and student learning 
within America’s Choice schools. These and 
other questions of interest will be the focus of 
CPRE’s ongoing exploration of the impact of 
America’s Choice on student learning. 
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Note: Other White = White students attending a non-America's Choice school; Other Min. = minority students attending a non-America's Choice school; 
AC White = White students attending an America's Choice schcx)l; AC Min. = minority students attending an America's Choice school. 
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